Update python3 to 3.7.6 #108

david · 2020-03-26T08:04:00Z

now a clean PR

add version

build form sources to get rid of the downgrades

pmenzel

Thank you for doing this upgrade. Some minor comments, and quite some questions about the reasoning of some of these changes, so it’s easier in the future to know if something is still needed or not.

A summary in the merge/pull request would also be nice. I guess the biggest change is to build Tensorflow and hic2cool from source?

pmenzel · 2020-03-26T09:03:32Z

python-3.7.6-0.build.sh

@@ -1,14 +1,14 @@
 #! /bin/bash

-COOKIE=$(mcookie); grep -v V_GREP_ME $0 > /dev/shm/runme-$COOKIE.sh ; sleep 1; exec bash /dev/shm/runme-$COOKIE.sh $1
+#COOKIE=$(mcookie); grep -v V_GREP_ME $0 > /dev/shm/runme-$COOKIE.sh ; sleep 1; exec bash /dev/shm/runme-$COOKIE.sh $1


Why is that not needed anymore? It’d be great if you mentioned that in the commit message (and also prefix it with python-3.7.6:). Maybe even delete that line.

is a dev line
copy the script to /dev/shm and run there
so you can edit while building

pmenzel · 2020-03-26T09:04:27Z

python-3.7.6-0.build.sh

@@ -175,7 +175,7 @@ make install
 python3 -m ensurepip
 pip3 install --prefix=$PREFIX -I pip

-pip3 install --prefix=$PREFIX ipython[all]
+pip3 install --upgrade --prefix=$PREFIX ipython[all]


Why is --ugprade added?

Oh, I have add this for rebuilding to find the downgrader

pmenzel · 2020-03-26T09:05:59Z

python-3.7.6-0.build.sh


 # pip3 install --prefix=$PREFIX HiCExplorer  # Would be a nice to have, but it likes to downgrade too much, pick some cherries
 pip3 install --prefix=$PREFIX HiCMatrix
-pip3 install --prefix=$PREFIX pyGenomeTracks
+pip3 install --prefix=$PREFIX pyGenomeTracks #downgrader cooler 0.8.7-> 0.8.5


Isn’t that also the version pinned in the hic2cooler requirements?

by hic2cool is the problem numpy and scipy and not cooler

pmenzel · 2020-03-26T09:06:56Z

python-3.7.6-0.build.sh

-    --cudart-lib-dir='${CUDA_ROOT}/lib64,${CUDA_ROOT}/lib64/stubs' \
-    --curand-lib-dir='${CUDA_ROOT}/lib64,${CUDA_ROOT}/lib64/stubs'\
-    --ldflags=-L$PREFIX/lib
+  	--cuda-enable-gl \


Please use tabs to indent this.

next time I promise :-)

pmenzel · 2020-03-26T09:07:17Z

python-3.7.6-0.build.sh

+	python3 setup.py install --prefix $PREFIX
+)
+
+# hic2cool from source to stop downgrading python packages


Which requirements are a problem?

numpy and scipy

pmenzel · 2020-03-26T09:08:16Z

python-3.7.6-0.build.sh

+
+#tensorflow
+(
+	TFVERSION=2.1.0


Why do we need to install Tensorflow from source? Why is the package not good enough?

f.e. the half of the cluster dont work with the pip package

pmenzel · 2020-03-26T09:08:44Z

python-3.7.6-0.build.sh

+	./configure
+	bazel build \
+		--cxxopt="-march=nehalem" \
+		--copt="-march=nehalem" \


Do all our servers support this?

No, but the important do it. This activate SSE4.1
this server f.e. dont support it
botanophobie
bromhidrosophobie
bugblatterbeastoftraal
carnophobie
deathofrats
deathofrats

if you want I can send you the full list

pmenzel · 2020-03-26T09:09:40Z

python-3.7.6-0.build.sh

+		//tensorflow:install_headers \
+		//tensorflow/tools/pip_package:build_pip_package
+
+	test -d tensorflow-pip && rm -rf tensorflow-pip


The check is not needed, as rm -rf will exit with success even if the directory does not exist.

I have changed it d0c427b

pmenzel · 2020-03-26T09:10:38Z

python-3.7.6-0.build.sh

+	#remove symlinks into TMPDIR
+	test -e tensorflow-pip.tar && rm tensorflow-pip.tar
+	tar -chf tensorflow-pip.tar tensorflow-pip
+	test -d tensorflow-pip && rm -rf tensorflow-pip


Ditto. The check is not needed.

thomas · 2020-03-26T09:52:22Z

Inline comments don't work either ...
I'm, referring to this one:

The check is not needed, as rm -rf will exit with success even if the directory does not exist.

If a directory doesn't exist, there is no need to try a removal, or?
Or if you like an efficiency based reason, testing within the shell beforehand will save a program call.
(I've heard such arguments before)

donald · 2020-03-26T17:55:49Z

Build on theinternet needed 6 hours :-) I've needed to remove

pip3 install --prefix=$PREFIX HiCMatrix
pip3 install --prefix=$PREFIX pyGenomeTracks #downgrader cooler 0.8.7-> 0.8.5

because pyGenomeTracks was updated on PyPi and not requires a version of HicMatrix which is not yet available on PyPi or some such nonsense.

thomas · 2020-03-26T18:03:00Z

The stuff from the HiC-clan is a real pita.
@david, would it be possible to install the 'former' version of pyGenomeTracks from source, or pin the version in the pip-install?

donald · 2020-03-26T18:06:36Z

When this was build into /package/pkg everything was fine. Until we change the system very basically, we'll always have the problem, that it is not reproducible because of PyPi updates. Thats not a fault of the this package. No change justified, imo.

updates unless we change something

donald · 2020-03-26T18:07:18Z

On which cuda machines should this tensorflow work? cpu was okay, but on "aros" I get
"tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version"

thomas · 2020-03-26T18:17:36Z

aros is using the former nvidia-driver (Driver Version: 418.56 CUDA Version: 10.1). You might try dalek (Driver Version: 440.44 CUDA Version: 10.2). Meaning the GPU version will only run if you are using a kernel that uses the 440.44 version (or above) of the NVIDIA driver.

donald · 2020-03-26T18:22:14Z

Works fine on dalek :-)

thomas · 2020-03-26T18:25:42Z

Dalek is the new GPU - beast. And in case you like Dr. Who, just scream 'eliminaaaaaaate' :)

pmenzel · 2020-03-28T06:08:28Z

hic2cool version 0.8.2 loosened the dependency requirements. I urge everyone to engage with upstream.

pmenzel · 2020-03-28T06:18:22Z

python-3.7.6-0.build.sh

+EOFHIC
+fi
+	pip3 install -r requirements.txt --prefix ${PREFIX}
+	python3 setup.py install --prefix ${PREFIX}


https://github.com/4dn-dcic/hic2cool/pull/44#issuecomment-605348278 claims, pip3 install is the recommended way to build and install packages.

david added 10 commits March 26, 2020 08:38

Python-3.7.6: add/update cudadir

e08dcf9

Python-3.7.6: change tmpdir bayel needs a lot of space

1118ca3

Python-3.7.6: remove tensorflow pip, build from source

8c742e4

Python-3.7.6: add comments to have eye on the downgraders

07c8642

Python-3.7.6: remove 'stage' part of boost, install is enough

484f0ef

Python-3.7.6: add multiqc (wish from someone)

c6f08d4

Python-3.7.6: pycuda

74e9dd0

add version

Python-3.7.6: hic2cool

7370c8c

build form sources to get rid of the downgrades

Python-3.7.6: build tensorflow from source

8e2a46f

comment magic cookie line

8ec8ae4

pmenzel reviewed Mar 26, 2020

View reviewed changes

Python-3.7.6: remove force by rm

d0c427b

pmenzel reviewed Mar 28, 2020

View reviewed changes

donald mentioned this pull request Mar 30, 2020

Update python3 to 3.7.7 donald #109

Merged

david closed this Apr 6, 2020

Update python3 to 3.7.6 #108

Update python3 to 3.7.6 #108

david commented Mar 26, 2020

pmenzel left a comment

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020 •

edited

pmenzel Mar 26, 2020

david Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

pmenzel Mar 26, 2020

david Mar 26, 2020

thomas commented Mar 26, 2020

donald commented Mar 26, 2020

thomas commented Mar 26, 2020

donald commented Mar 26, 2020

donald commented Mar 26, 2020

thomas commented Mar 26, 2020

donald commented Mar 26, 2020 •

edited

thomas commented Mar 26, 2020

pmenzel commented Mar 28, 2020

pmenzel Mar 28, 2020

Update python3 to 3.7.6 #108

Update python3 to 3.7.6 #108

Conversation

david commented Mar 26, 2020

pmenzel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david Mar 26, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomas commented Mar 26, 2020

donald commented Mar 26, 2020

thomas commented Mar 26, 2020

donald commented Mar 26, 2020

donald commented Mar 26, 2020

thomas commented Mar 26, 2020

donald commented Mar 26, 2020 • edited

thomas commented Mar 26, 2020

pmenzel commented Mar 28, 2020

Choose a reason for hiding this comment

david Mar 26, 2020 •

edited

donald commented Mar 26, 2020 •

edited